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Three-dimensional (3D) variants of the Kitaev model can harbor gapless spin liquids with a Majorana Fermi 
surface on certain tricoordinated lattice structures such as the recently introduced hyperoctagon lattice. Here we 
investigate Fermi surface instabilities arising from additional spin exchange terms (such as a Heisenberg cou¬ 
pling) which introduce interactions between the emergent Majorana fermion degrees of freedom. We show that 
independent of the sign and structure of the interactions, the Majorana surface is always unstable. Generically 
the system spontaneously doubles its unit cell at exponentially small temperatures and forms a spin liquid with 
line nodes. Depending on the microscopies further symmetries of the system can be broken at this transition. 
These spin-Peierls instabilities of a 3D spin liquid are closely related to BCS instabilities of fermions. 


The interplay of frustration and correlations in quantum 
magnets engenders a rich variety of quantum states with frac¬ 
tional excitations that are collectively referred to as quan¬ 
tum spin liquids (QSL) jTJ. Archetypal instances of such 
states include gapped topological QSLs with anyonic quasi¬ 
particle excitations as well as gapless QSLs where emergent 
spinon excitations form nodal structures such as Dirac points, 
Fermi lines or Fermi surfaces reminiscent of metallic states. 
A family of such gapless QSLs is realized in two- and three- 
dimensional variants of the Kitaev model 0 , in which a high 
level of exchange-frustration is induced by competing bond- 
directional interactions of the form 

-^Kitaev = J X a i a l • (1) 

7—links 

Here spin- 1/2 moments a on sites i and j are coupled 
via an Ising-like exchange whose easy-axis aligns with the 
7 = x, y, z orientation of the bonds of the underlying tri¬ 
coordinated lattices. Kitaev’s seminal solution 0 of this 
model for the honeycomb lattice allows to analytically track 
the fractionalization of the original spin-1/2 moments into 
emergent massless Majorana fermions (spinons) and massive 
J-2 gauge excitations (visons). The collective QSL ground 
state is a (semi)metallic state formed by the itinerant Ma¬ 
jorana fermions. The qualitative nature of this Majorana 
metal turns out to depend on the underlying lattice: for the 
two-dimensional honeycomb lattice the itinerant Majorana 
fermions form two Dirac cones 0 (akin to the well-known 
electronic band structure of graphene), while for the three- 
dimensional hyperhoneycomb and hyperoctagon lattices the 
gapless Majorana modes form a Fermi line J 3 | and an en¬ 
tire Fermi surface a, respectively. In the presence of addi¬ 
tional time-reversal symmetry breaking terms the Majorana 
fermions (on the hyperhoneycomb lattice) can even form a 
topological semimetal with Weyl nodes 0. Interest in such 
three-dimensional Kitaev models has been sparked by the re¬ 
cent experimental observation of spin-orbit entangled j = 1/2 
Mott insulators with strong bond-directional interactions of 
the form 0 in the iridates (/ 3 ,7)-Li2lrC>3 || 6 ]-[ 9 ) , where the 
iridium sites form three-dimensional, tricoordinated lattice 
structures. The synthesis of such three-dimensional Kitaev 
structures expands an intense ongoing search for solid-state 


realizations of the original two-dimensional Kitaev model, 
which - following the early theoretical guidance of Khaliullin 
and coworkers m - has put materials such the layered iri¬ 
dates Na2lrC>3, a-LQIrOa OH and more recently o-RuCI - 
m into focus. 


In this paper, we will concentrate on three-dimensional 
Kitaev spin liquids characterized by Majorana Fermi sur¬ 
faces and show that they generically dimerize, i.e. dou¬ 
ble their unit cell at low temperature. This instability can 
be viewed as a generalization of the spin-Peierls transition 
in one-dimensional systems El El, or more generally, of 
the tendency of frustrated low-dimensional spin systems to 
form valence-bond solids 031 . The spin-Peierls instability of 
(quasi-) one-dimensional spin systems describes that an arbi¬ 
trarily small coupling of a spin chain to classical lattice de¬ 
grees of freedom leads at low temperature to a dimerization 
and a gap in the spin system El El: the energy gain by 
opening the gap is larger than the energy needed to distort the 
lattice. When the phonon mode is, however, treated quantum 
mechanically, a dimerization occurs only when a critical cou¬ 
pling strength is reached EE As we will show, for the 3 D Ki¬ 
taev spin liquid a variant of the spin-Peierls transition occurs 
at low T even in the absence of lattice degrees of freedom and 
for arbitrarily weak perturbations. Notably, the result of this 
instability is not a short-range valence-bond ordered state, but 
still a QSL - yet one, in which the original Majorana Fermi 
surface has collapsed into a line of gapless modes. 


Model system - For concreteness, we will focus our analysis 
on a specific 3 D Kitaev model and return to a more general 
discussion later. We look at the Kitaev model defined on the 
so-called hyperoctagon lattice depicted in Fig. |T] For this sys¬ 
tem a gapless QSL ground state with a Majorana Fermi sur¬ 
face was established in Ref. [ 4 ] to which we also refer for tech¬ 
nical details of the analytical solution of this model. In brief, 
the spin degrees of freedom fractionalize to itinerant Majo¬ 
rana fermions interacting with static J-2 gauge fields 0 . In 
the (flux-free) ground state sector of the Z2 gauge field, the 
original spin model ([T} reduces to a free hopping model of 
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FIG. 1. (color online) The hyperoctagon lattice is a tricoordinated 
bcc lattice without inversion symmetry (space group I4i32) and a 
four-site unit cell (with atom positions 1,..., 4 and unit cell vec¬ 
tors ai, ao and a;; indicated in the figure). For the definition of the 
Kitaev model on this lattice we define nearest neighbor spin interac¬ 
tions of the a x a x , a v a v , and a z a z type assigned to the green, red, 
and blue bonds, respectively. The vectors ai, a) = a2 — a3, and 
a 3 = — a ] + a 2 + a 3 are the lattice translation vectors of each of the 
two sublattices (denoted by white and grey lattice sites). 


Majorana fermions 

Hq = -iJ Y, ci(R)c 3 (R - a 2 ) + c 2 (R)c 4 (R - a 3 ) 

R 

+ ci(R)c 2 (R) - c 3 (R)c 4 (R) - c 2 (R)c 3 (R) 

+ Ci(R)c 4 (R —a 4 ). (2) 

Here R is the position of the unit cells within the bcc Bravais 
lattice (see Fig.jT]), each containing four sites labeled by i = 
1,2, 3,4; ay are the lattice vectors defined in Fig. [T] and the 
Majorana operators obey the usual algebra {cj(R), Cj (R')| = 
2<%<5r,r'. 

Diagonalizing the Hamiltonian ([2]), one finds that two of 
the four bands have a gap of order J; the other two bands be¬ 
come gapless on two two-dimensional surfaces in momentum 
space defined by the equation cos k x + cos k y + cos k z = — | 
0). The two closed Majorana surfaces are centered around 
the momenta ±k 0 /2 with k 0 = 27 t(1, 1,1) as illustrated in 
Fig. 0a). Note that only 2ko is a unit vector of the reciprocal 
lattice, but not ko itself. Most importantly, there is a perfect 
nesting condition between the two Majorana surfaces, 


e k — £k+k 0 j (3) 

which is not specific to the microscopic Hamiltonian (jTj) but 
arises from the peculiar form of time-reversal symmetry T of 
the underlying spin model realized by HU SI 

Cy(R)^ (— l)V k ° R c,(R). (4) 

For the following discussion it is convenient to combine the 
two low-energy Majorana bands into a single band of a (com¬ 
plex) fermion, which allows to use the more familiar language 


a) Majorana fermions 



b) complex fermion 



FIG. 2. (color online) a) Sketch of the two Majorana Fermi surfaces 
in the Brillouin zone. In the Majorana language, the Peierls instabil¬ 
ity is due to perfect nesting between the two surfaces, b) The two 
Majorana surfaces combine to a single Fermi surface of a (complex) 
fermion. In the fermionic language, the Peierls instability is the usual 
BCS instability. 


of superconductors to describe the system. To do so, we de¬ 
note the low-energy Majorana excitations with momentum k 
by 7^ 7 2 with e\, < e^. Note that the particle-hole symmetry of 
Majorana fermions readily implies 71 (k) = 7 ^—k), which 
requires to either restrict the discussion to half the Brillouin 
zone or, alternatively, to the upper energy band. Doing the lat¬ 
ter, we introduce the Fermion operator /k by /k = ^75 7k- 
This reformulation combines the two Majorana surfaces in 
Fig-0 a) into a single Fermi surface of a complex fermion 
centered around k 0 /2, see Fig. |2]b). In terms of the complex 
fermion, time-reversal symmetry 0 becomes 


ft r > ft 

■'kn/ 2 +k 7 J\ 


k 0 / 2 -k > 


(5) 


constraining the fermionic energy spectrum to be symmetric 
relative to ko/ 2 : 


F'ko/2+k — -®ko/2-k • (6) 

This energy relation naturally leads to consider pairing terms 
of the form /k 0 / 2 +k/k 0 / 2 -k- However, such pairs carry a fi¬ 
nite momentum ko El and can therefore only arise in a phase 
in which translational symmetry is spontaneously broken. In 
the following, we will show that such a spontaneous symme¬ 
try breaking will occur in the presence of additional spin ex¬ 
change terms. 

Interactions.- Any deviation from the ideal Kitaev model, 
e.g. by introducing spin-spin interactions of the Heisenberg 
form, Dzyaloshinskii Moriya interactions, three-spin interac¬ 
tions, etc., will induce interactions of the Majorana fermions. 
As long as the size of these perturbations is sufficiently small 
(compared to the Kitaev interaction J, which in particular sets 
the energy scale for the flux gap), the interaction can be writ¬ 
ten in terms of the Majorana fermions only (without any con¬ 
tributions from the fluxes) and will remain short-ranged. For 
the hyperoctagon lattice, two types of Majorana interactions 
turn out to be the most local ones, which are therefore also ex¬ 
pected to be the most important ones in an expansion around 
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the Kitaev model lfl8ll . We parametrize them by their overall where the odd order parameter, Ak = — A_ k- is computed 
strength U and an angle a as from the mean-field equation 


-Hint = ~U( CQSCty^ Ci(R)c 2 (R)C3(R)C!(R + a 2 ) 

R 

+ sin a ci(R)c 2 (R)c 3 (R)c 4 (R) + sym.) . (7) 

R 

Note that each term in the above expression is only a single 
representative of 24 distinct terms obtained by the 48 symme¬ 
try transformations of the I4i32 symmetry group of the hype¬ 
roctagon lattice 03 . When adding, e.g., a nearest-neighbor 
Heisenberg coupling Jh to the Kitaev model, one expects 
U ~ Jfj/J 3 0. In general, the sizes of U and a will de¬ 
pend on all types of microscopic details and are difficult to 
predict quantitatively. We therefore analyze the effect of the 
interactions 0 for varying values of a (and fixed, small U). 

To analyze // (l + H uit for small U, we first project the inter¬ 
action |T|! onto the low-energy degrees of freedom described 
by the complex / fermions introduced above and obtain 

HeS = T e k/k/k + /k 3 /k 2 /k 3 /k!+k 2 -k 3 

k ki,k 2 ,k 3 

+ X! ^ / k 1 .k 2 ,k 3 /ki/k 2 /k 3 k 3 —k 2 —k 3 A h.C. 
ki,k 2 ,k 3 

( 8 ) 

The momentum dependence of V 1,2 can easily be obtained 
numerically using Eq. |T}, the eigenmodes of TTo, and the def¬ 
inition given above. Note that symmetry-allowed terms of 
the form f*ft k or ftjt 2 ft 3 f kl+k2+k3 do not contribute to 
the low-energy sector as at least one of the momenta is far 
away from the Fermi surface. 

Pairing instability.- As // (l can be written as non-interacting 
spinless fermions, it is not surprising that the pairing insta¬ 
bility due to terms such as /k 0 / 2 +k/k 0 / 2 -k' s governed by 
the same type of logarithms which are responsible for p- wave 
superconductivity ED. For small U , one can therefore ex¬ 
pect that dimerization sets in below a transition temperature 
T c with 


T c = E 0 , a e~ c ‘* J/u for t/<J, (9) 

where /i (l o is an energy scale of order J. The dimensionless 
constant c Q can be computed exactly from a one-loop renor¬ 
malization group calculation or, alternatively, from a BCS 
mean-field calculation. In the following we will use BCS the¬ 
ory to compute T c and the structure of the order parameter. 
Our approach allows to calculate c a exactly, but unfortunately 
not the prefactor h'o,, |20l . 

To describe the dimerized phase, we consider the BCS-style 
Hamiltonian 


HbCS — e fc/k/k + ^ k /k 0 /2+k/k 0 /2-k + ^-G- , 

k k x > 0 

( 10 ) 


^k — 2 y^l / k 1 n/2+k.kn/2-k.kn/2+a(/ko/2+q/ko/2-q) (H) 

q 

+ 1 J ^ / ko/2-|-k.ko/2—k,ko/2+q('^*ko/2-t-q'^ > ko/2—q) 


where we assumed that the interactions in Eq. | 8 } have been 
completely antisymmetrized with respect to the fermionic op¬ 
erators. All expectation values are computed with Hbcs- 
Note that in contrast to the standard BCS theory there is no 
(7(1) symmetry. It turns out that time-reversal symmetry 0 
is not spontaneously broken which leads to a purely imaginary 
Ak, A() = - A k . The latter, together with Ak = - A al¬ 
ready implies that pairing cannot gap the Fermi surface com¬ 
pletely, but only reduce it to an odd number of lines. 

To simplify Eq. CD and to solve it with high nu¬ 
merical precision to leading logarithmic order, we rewrite 
the q integration into an integral on the Fermi surface 
and an energy integration perpendicular to it, f d 3 q = 
/ FS d 2 kp f N kp {e)de. We furthermore approximate the di¬ 
rectional dependent density of state by its value of the Fermi 
surface N kr ( 0) = ( 27 r) 3 J F(kF ) for -2J < e < 2J. 
Similarly, we approximate both Ak and the matrix ele¬ 
ments of V 1 and V 2 by their values on the Fermi surface. 
This allows for an accurate evaluation of the e integration, 

r 1 / n f \ A p2J , tanh yj e 2 + | A qF | 2 /(2 T) 

J de(fk 0 /2+qfko/2-q) ~ Aq F J Q de 

The last term is evaluated numerically but it is also well 

(1.762 T) 2 ] . For the 


described by A qF In 


4J/^A^F 


plots of this paper, we discretize the Fermi surface using a 
total of 1344 patches. 

Phase diagram - Using the largest eigenvalue of the lin¬ 
earized BCS equation CD- one can directly determine c a in 
Eq.0 with Eq j01 « 2.27 J within our cutoff scheme. l/c Q is 
plotted in the upper panel of Fig. [3] For 0 < a < 4.65 (red) 
the eigenvalue is three-fold degenerate. The three eigenvec¬ 
tors form an irreducible representation of the point group O 
and transform like k x , k y and k z (T) representation, see sup¬ 
plemental material). For 4.65 < a < 2 tt (green), in contrast, 
the eigenvalue is unique and the eigenvector transforms like 
k x k y k z (.4 2 representation). Analyzing the BCS equation be¬ 
yond the linear approximation, one obtains the phase diagram 
shown for U = 0.4J in the middle panel of Fig. [3]as function 
of temperature T and a. For other values of U one obtains 
very similar phase diagrams, but all transition temperatures 
and the ratio of the largest and smallest T c change exponen¬ 
tially according to Eq. 0 . 

We find five different phases, denoted A E in Fig. [3] 
which are distinguished by symmetry and/or the number of 
line nodes; the corresponding distribution of order parameters 
is shown in the lower panel of Fig. [3] In table[I]an overview of 
the properties of the symmetry-broken phases is given. In all 
phases the unit-cell has doubled. In phase E this is the only 
broken symmetry. In the phases A — D besides the transla¬ 
tional symmetry also other lattice symmetries are broken. As 
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FIG. 3. (color online) Upper figure: The largest eigenvalue of the 
linearized BCS equation determines c a in Eq. & Middle fig¬ 
ure: Phase diagram for U = 0.4J as function of the angle a. 
There are 4 different phases distinguished by symmetry (see ta¬ 
ble [TJi. Lower figure: For the five phases A-E the angular distribu¬ 
tion of the order parameter A, on the Majorana surface is shown 
(a = 0.5-7T, 0.77T, 7r, 1.357T, 1.487T, 1.757 t). While there is a single 
line node in phase A, there are three non-crossing line nodes for B, 
C, D, and three line nodes with 6 crossing points for E. 



order parameter 

point group 

domains 

line nodes 

A, A’ 

B 

k X + ky ky 

D 3 (32) 

8 

1 

3 

C 

kx 

D 4 (422) 

6 

3 

D 

cik x + C 2 k y 

C 2 (2) 

24 

3 

E 

kx kykz 

O (432) 

2 

3 (crossing) 


TABLE I. Overview of the symmetry broken phases, see Fig. [3] In 
all phases the unit cell is doubled. The first column shows schemati¬ 
cally the symmetry of the order parameter (momenta measured from 
ko/2). Furthermore, the point group of the symmetry broken phase, 
the number of domains and the number of line nodes are tabulated. 

at T c the eigenvalue c a is three-fold degenerate, the phase di¬ 
agram can be understood best in terms of a three-component 
order parameter, < 1 > = (< 1 >|, $ 2 , $ 3 ) where <!>, corresponds to 
an eigenvector transforming like fcj. Close to T c , one can use 
a Landau description of the free energy of the form 

F » (T—T c )$ 2 + 5 ($ 2 ) 2 +ff , (^+^+^)+0($ 6 ). (12) 
It predicts that just below T c the order parameter points for 


g' > 0 in one of the (111) directions (phases A, A’, and B 
colored in red in Fig. [3}. The order parameter therefore has 
a 120 °-rotation symmetry around this axis, see also the lower 
panel in Fig. [3] In the crystal structure this (111) direction 
will be shortened or elongated (depending on the sign of the 
coupling to the lattice). In total the number of domains is 
8 , corresponding to four possible structural transformations 
times two different translational domains. 

For g' < 0, in contrast, the order parameter points along 
a (100) direction (phase C with 6 domains) and the lattice 
will shrink or expand along that direction. Interestingly, it 
turns out that all phases besides the phase A are character¬ 
ized by three line nodes instead of a single one, see Fig. [3] 
In the language of superconductivity, one would call this an 
“extended /v-wave order parameter” |[22l . The simple Landau 
theory ( [T2| can fail at low T and indeed phase 1) cannot be 
described by terms up to order <1> 4 . Here further symmetries 
are broken (see table|I|, $ points in a low-symmetry direction, 
e.g., (ci, C 2 , 0), and the ordered phase is characterized by 24 
different domains. In this case the only remaining symmetry 
of the order parameter is that it changes sign by 180°-rotation 
around the ( 001 ) axis. 

Experimental signatures.- The spin-Peierls instability of 3D 
Kitaev spin liquids with a Majorana Fermi surface and its re¬ 
sulting nodal-line QSL provide a distinct experimental fin¬ 
gerprint, which could facilitate the ongoing search for Ki¬ 
taev spin liquids in spin-orbit entangled j = 1/2 Mott in¬ 
sulators. In thermodynamics, one expects to observe (i) the 
absence of magnetic order breaking time-reversal symmetry, 
(ii) an approximately constant specific heat coefficient c v /T 
over a wide intermediate temperature range (indicative of the 
Majorana Fermi surface), (iii) a structural phase transition 
with unit-cell doubling at small temperatures T c , and (iv) a 
low-temperature specific heat with c v /T « T (indicative of 
the nodal lines) with possible logarithmic corrections (arising 
from the crossing points of the nodal lines for phase E). Be¬ 
sides the structural phase transition discussed in this publica¬ 
tion, three dimensional Z 2 gauge theories (without monopoles 
Pfl ) also show a finite-T phase transition 1231124 ! of (inverted) 
Ising universality class. Note that we have implicitly assumed 
in our calculation that the T c of the structural transition is well 
below this transition temperature where flux lines proliferate. 
Summary- We have shown that a time-reversal symmetric 
spin liquid with Majorana surface is always unstable and 
spontaneously undergoes a spin-Peierls transition to a nodal 
QSL at low temperatures. We expect that this is a generic 
property of Majorana surfaces. To show this, it is useful to 
classify time-reversal invariant Majorana systems by the value 
of the vector k 0 characterizing the time-reversal operation in 
Eq. (|4| or, equivalently, Eq. (|5]». If ko vanishes, terms of the 
form /k 0 / 2 +kJk 0 / 2 -k occur even in the absence of symmetry 
breaking and - instead of a Fermi surface - only a state with 
nodal lines forms 0E3. Fermi surfaces therefore exist only 
for finite k 0 . In this case, however, time-reversal invariance 
guarantees the existence of a BCS-type instability. As any 
interaction of Majorana excitations always involves four dif- 
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ferent sites, the momentum-dependent interaction will always 
have an attractive channel leading to a transition where Ma- 
jorana pairs condensate at finite momentum ko. We therefore 
expect that Majorana surfaces can only survive for T —J 0 in 
cases where time-reversal symmetry is broken either sponta¬ 
neously or explicitly, e.g. by an external magnetic field - a 
situation which we plan to investigate in the future. 

Acknowledgments.- M.H. acknowledges partial support 
through the Emmy-Noether program of the DFG. 


[1] For a recent review see e.g. L. Balents, Spin liquids in frustrated 
magnets. Nature 464 , 199 (2010). 

[2] A. Kitaev, Anyons in an exactly solved model and beyond, Ann. 
Phys. 321 , 2 (2006). 

[3] S. Mandal and N. Surendran, Exactly solvable Kitaev model in 
three dimensions, Phys. Rev. B 79 . 024426 (2009). 

[4] M. Hermanns and S. Trebst, Quantum spin liquid with a Majo¬ 
rana Fermi surface on the three-dimensional hyperoctagon lat¬ 
tice, Phys. Rev. B 89, 235102 (2014). 

[5] M. Hermanns, K. O’Brien, and S. Trebst, Weyl Spin Liquids, 
Phys. Rev. Lett. 114 , 157202 (2015) 

[6] T. Takayama, A. Kato, R. Dinnebier, J. Nuss, and H. Takagi, 
Hyper-honeycomb iridate /S-LLIrO.i as a platform for Kitaev 
magnetism, Phys. Rev. Lett. 114, 077202 (2015) 

[7] K. A. Modic, T. E. Smidt, I. Kimchi, N. P. Breznay, A. Biffin, 
S. Choi, R. D. Johnson. R. Coidea, P. Watkins- Curry, G. T. 
McCandless, J. Y. Chan, F. Gandara, Z. Islam, A. Vishwanath, 
A. Shekhter, R. D. McDonald, and J. G. Analytis, Realization 
of a three-dimensional spin-anisotropic harmonic honeycomb 
iridate. Nature Comm. 5, 4203 (2014) 

[8] A. Biffin, R. D. Johnson, Sungkyun Choi, F. Freund, S. Manni, 
A. Bombardi, P. Manuel. P. Gegenwart, and R. Coidea, Uncon¬ 
ventional magnetic order on the hyperhoneycomb Kitaev lattice 
in /S-LLIrCH: Full solution via magnetic resonant x-ray diffrac¬ 
tion Phys. Rev. B. 90 , 205116 (2014) 

[9] A. Biffin, R. D. Johnson, I. Kimchi, R. Morris, A. Bombardi, J. 
G. Analytis, A. Vishwanath, and R. Coidea, Noncoplanar and 
Counterrotating Incommensurate Magnetic Order Stabilized by 
Kitaev Interactions in y-LLIrCL Phys. Rev. Lett. 113 , 197201 
(2014) 

[10] G. Khaliullin, Orbital Order and Fluctuations in Mott Insula¬ 
tors, Prog. Theor. Phys. Suppl. 160 , 155 (2005); G. Jackeli 
and G. Khaliullin, Mott Insulators in the Strong Spin-Orbit 
Coupling Limit: From Heisenberg to a Quantum Compass 
and Kitaev Models, Phys. Rev. Lett. 102 , 017205 (2009); J. 
Chaloupka, G. Jackeli, and G. Khaliullin, Kitaev-Heisenberg 
Model on a Honeycomb Lattice: Possible Exotic Phases in Irid¬ 
ium Oxides A 2 lr 03 , Phys. Rev. Lett. 105 , 027204 (2010) 

[11] Yogesh Singh, S. Manni, J. Reuther, T. Berlijn, R. Thomale, W. 
Ku, S. Trebst, and P. Gegenwart, Relevance of the Heisenberg - 
Kitaev Model for the Honeycomb Lattice Iridates A 2 lr 03 , 
Phys. Rev. Lett. 108 , 127203 (2012) S. K. Choi et al. Spin 
Waves and Revised Crystal Structure of Honeycomb Iridate 
Na 2 Ir0 3 , Phys. Rev. Lett. 108 , 127204 (2012); R. Comin et al ., 
Na 2 lr 03 as a Novel Relativistic Mott Insulator with a 340-meV 
Gap, Phys. Rev. Lett. 109 , 266406 (2012); Feng Ye, Songxue 
Chi, Huibo Cao, Bryan C. Chakoumakos, Jaime A. Femandez- 
Baca, Radu Custelcean, T. F. Qi, O. B. Korneta, and G. Cao, Di¬ 
rect evidence of a zigzag spin-chain structure in the honeycomb 


lattice: A neutron and x-ray diffraction investigation of single¬ 
crystal Na 2 Ir0 3 , Phys. Rev. B 85 , 180403 (2012), H. Gretars- 
son et al., Crystal-Field Splitting and Correlation Effect on the 
Electronic Structure of A 2 lrC> 3 , Phys. Rev. Lett. 110 , 076402 
(2013), H. Gretarsson et al.. Magnetic excitation spectrum of 
Na 2 lrC >3 probed with resonant inelastic x-ray scattering, Phys. 
Rev. B 87 , 220407 (2013); Sae Hwan Chun, Jong-Woo Kim, 
Jungho Kim, H. Zheng, Constantinos C. Stoumpos, C. D. Malli- 
akas, J. F. Mitchell, Kavita Mehlawat, Yogesh Singh, Y. Choi, 
T. Gog, A. Al-Zein, M. Moretti Sala, M. Krisch, J. Chaloupka, 

G. Jackeli, G. Khaliullin, and B. J. Kim, Direct Evidence for 
Dominant Bond-directional Interactions in a Honeycomb Lat¬ 
tice Iridate Na 2 lrC> 3 , Nature Physics, advance online publica¬ 
tion (2015) 

[12] K. W. Plumb, J. P. Clancy, L. J. Sandilands, V. Vijay Shank a r, Y. 
F. Hu, K. S. Burch, Hae-Young Kee, and Young-June Kim, ct- 
RUCI 3 : A spin-orbit assisted Mott insulator on a honeycomb 
lattice, Phys. Rev. B 90 , 041112(R) (2014); Luke J. Sandi¬ 
lands, Yao Tian, K. W. Plumb, Young-June Kim, and Ken¬ 
neth S. Burch, Scattering Continuum and Possible Fraction¬ 
alized Excitations in rv-RuCL, Phys. Rev. Lett. 114 , 147201 
(2015), V. Vijay Shankar, Heung-Sik Kim, and Hae-Young 
Kee, Kitaev magnetism in honeycomb RuCL with intermedi¬ 
ate spin-orbit coupling, arXiv: 1411.6623; M. Majumder, M. 
Schmidt, H. Rosner, A. A. Tsirlin, H. Yasuoka, and M. Baenitz, 
Anisotropic Ru3+ 4d5 magnetism in the alpha-RuC13 honey¬ 
comb system: susceptibility, specific heat and Zero field NMR, 
arXiv:1411.6515, L. J. Sandilands, Y. Tian, A. A. Reijnders, 

H. -S. Kim, K. W. Plumb, H.-Y. Kee, Y.-J. Kim, and K. S. 
Burch, Orbital excitations in the 4d spin-orbit coupled Mott 
insulator a-RuCfs, arXiv: 1503.07593, Y. Kubota, H. Tanaka, 
T. Ono, Y. Narumi, and K. Kindo, Successive magnetic phase 
transitions in q-RuCL : XY-like frustrated magnet on the hon¬ 
eycomb lattice arXiv:1503.03591 A. Banerjee, C.A. Bridges, 
J-Q. Yan, A.A. Aczel, L. Li, M.B. Stone, G.E. Granroth, M.D. 
Lumsden, Y. Yiu, J. Knolle, D.L. Kovrizhin, S. Bhattacharjee, 
R. Moessner, D.A. Tennant, D.G. Mandrus, and S.E. Nagler, 
Proximate Kitaev Quantum Spin Liquid Behaviour in o -RuCL,, 
arXiv: 1504.08037 

[13] E. Pytte, Peierls instability in Heisenberg chains, Phys. Rev. B 
10 . 4637 (1974) 

[14] M. C. Cross and D. S. Fisher, A new theory of the spin- 
Peierls transition with special relevance to the experiments on 
TTFCuBDT, Phys. Rev. B 19, 402 (1979) 

[15] N. Read and S. Sachdev, Spin-Peierls, valence-bond solid, and 
Neel ground states of low-dimensional quantum antiferromag- 
nets, Phys. Rev. B 42, 4568 (1990) 

[16] G. S. Uhrig, Nonadiabatic approach to spin-Peierls transitions 
via How equations, Phys. Rev. B 57, R14004 (1998), A. W. 
Sandvik and D. K. Campbell, Spin-Peierls Transition in the 
Heisenberg Chain with Finite-Frequency Phonons, Phys. Rev. 
Lett. 83, 195 (1999) 

[17] Note that for other three-dimensional, tricoordinated lattices 
(such as the hyperhoneycomb lattice) ko actually vanishes. For 
these lattices, these pairing terms are expected to occur even 
in the absence of spontaneous symmetry breaking. As a conse¬ 
quence, none of the QSL ground states of these lattices exhibit 
a Majorana Fermi surface but a nodal line structure instead (3). 

[18] A third type of interaction term involving a Majorana and its 
three neighboring sites (e.g., ci(R)c 2 (R)c 3 (R)c 4 (R — 83 )) 
turns out to be forbidden by time reversal symmetry. 

[19] As explained in detail in the supplement, one has to ensure 
by a combination of real-space and gauge transformations that 
one remains in the same (free-flux) gauge sector (of the ground 



6 


state) when implementing the symmetries. 

[20] For our calculation we assume that T c remains well below the 
gap for flux-loop excitations, which we do not include in our 
treatment. The numerical value of the flux gap is sa 0.18J for 
the hyperoctagon model at hand. Note that some of the T c val¬ 
ues in the middle panel of Fig.[3]overshoot this value due to the 
relatively large value of U chosen for this illustration. 

[21] Anthony J. Leggett, A theoretical description of the new phases 
of liquid 3 He, Rev. Mod. Phys. 47 , 331 (1975) 

[22] L. Mathey, S.-W. Tsai, and A. H. Castro Neto, Exotic supercon¬ 


ducting phases of ultracold atom mixtures on triangular lattices, 

Phys. Rev. B 75. 174516 (2007) 

[23] J. Nasu, M. Udagawa, and Y. Motome, Vaporization of Kitaev 
Spin Liquids, Phys. Rev. Lett. 113 , 197205 (2014) 

[24] I. Kimchi, J. G. Analytis, and A. Vishwanath, Three- 
dimensional quantum spin liquids in models of harmonic- 
honeycomb iridates and phase diagram in an infinite-D approx¬ 
imation, Phys. Rev. B 90 , 205126 (2014) 


Symmetry transformations 

Following Kitaev’s original approach El we can solve the Hamiltonian (|T]» by representing the spins in terms of four Majorana 
fermions 


and writing ([TJ) 


ia^Cj 


(13) 


T^Kitaev — J ^ , 

7—links 


(14) 


where we defined bond operators u 1 - ■, = iajaj,. By mapping the spin degree of freedom to four Majorana fermions, we 
introduced a Z 2 gauge degree of freedom on each bond and the physical space is defined by a^a^a^Cj |phys) = |phys). Following 
Kitaev, we fix a gauge to do our computations, see Ref. SI for details on the gauge choice. As a result, lattice symmetries as 
well as time-reversal symmetry may have to be supplemented with a gauge transformation in order to be a symmetry of the 
gauge-fixed Hamiltonian. Table [Ul] lists the appropriate gauge transformation for each of the 48 symmetry transformations of 
the 74! 32 space group, implemented by 


c q (R) —> c a (R)cr a ,<5 k °' R . (15) 


Here cry, j = 1,..., 4 denote the sign factor for the Majorana operator at position j - listed in the second column of Table 
The values of S = ±1 is denoted in the third column. 


Ill 
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coordinate gauge transformations staggered coordinate gauge transformations staggered 


X 

y 

z 

111 1 

1 

1 / 2 + x 

1 / 2 + y 

1 / 2 + z 

111 1 

1 

1 / 2 -x 

-y 

1 / 2 + z 

11-1 -1 

1 

-X 

1 / 2 -y 

z 

11-1 -1 

1 

-X 

1 / 2 + y 

1/2 -z 

1-11 -1 

1 

1 / 2 -x 

y 

-z 

1-11 -1 

1 

1 / 2 + x 

1 / 2 -y 

-z 

1 - 1-1 1 

1 

X 

-y 

1 / 2 -z 

1 - 1-1 1 

1 

z 

X 

y 

111 1 

1 

1/2 + z 

1 / 2 + x 

1 / 2 + y 

111 1 

1 

1 / 2 + z 

1 / 2 -x 

-y 

1 - 1-1 1 

1 

z 

-X 

1 / 2 -y 

1 - 1-1 1 

1 

1/2 -z 

-X 

1 / 2 + y 

11-1 -1 

1 

-z 

1 / 2 -x 

y 

11-1 -1 

1 

-z 

1 / 2 + x 

1 / 2 -y 

1-11 -1 

1 

1 / 2 -z 

X 

-y 

1-11 -1 

1 

y 

z 

X 

111 1 

1 

1 / 2 + y 

1/2 + z 

1 / 2 + x 

111 1 

1 

-y 

1 / 2 + z 

1 / 2 -x 

1-11 -1 

1 

1 / 2 -y 

z 

-X 

1-11 -1 

1 

1 / 2 + y 

1 / 2 -z 

-X 

1 - 1-1 1 

1 

y 

-z 

1 / 2 -x 

1 - 1-1 1 

1 

1 / 2 -y 

-z 

1 / 2 + x 

11-1 -1 

1 

-y 

1 / 2 -z 

X 

11-1 -1 

1 

3/4+ y 1/4 + x 

1/4-z 

1 -1 -1 -1 

-1 

11/4 + y 3/4 + x 

3/4-z 

1 -1 -1 -1 

-1 

3/4-y 

3/4-x 

3/4-z 

11-1 1 

-1 

1/4 - y 

1/4-x 

1/4-z 

11-1 1 

-1 

1/4+ y 

1/4-x 

3/4+ z 

1-11 1 

-1 

3/4+ y 3/4-x 

1/4+ z 

1-11 1 

-1 

1/4-y 

3/4+ x 

1/4+ z 

111 -1 

-1 

3/4-y 

1/4+ x 

3/4+ z 

111 -1 

-1 

3/4+ x 

1/4+ z 

1/4-y 

1 -1 -1 -1 

-1 

1/4+ x 

3/4+ z 

3/4-y 

1 -1 -1 -1 

-1 

1/4-x 

3/4+ z 1/4+ y 

111 -1 

-1 

3/4-x 

1/4+ z 3/4+ y 

111 -1 

-1 

3/4-x 

3/4-z 

3/4-y 

11-1 1 

-1 

1/4-x 

1/4-z 

1/4-y 

11-1 1 

-1 

1/4+ x 

1/4-z 

3/4+ y 

1-11 1 

-1 

3/4+ x 

3/4-z 

1/4+ y 

1-11 1 

-1 

3/4+ z 1/4+ y 

1/4-x 

1 -1 -1 -1 

-1 

1/4 + z 3/4 + y 

3/4-x 

1 -1 -1 -1 

-1 

1/4+ z 

1/4-y 

3/4+ x 

1-11 1 

-1 

3/4+ z 

3/4-y 

1/4+ x 

1-11 1 

-1 

1/4-z 

3/4 + y 1/4+ x 

111 -1 

-1 

3/4-z 

1/4 + y 3/4 + x 

111 -1 

-1 

3/4-z 

3/4-y 

3/4-x 

11-1 1 

-1 

1/4-z 

1/4-y 

1/4-x 

11-1 1 

-1 


TABLE II. Overview of the 48 symmetry transformations of the I4i32 symmetry group of the hyperoctagon lattice and the associated gauge 
transformation in the Majorana model. The left column displays the symmetry transformation in real space. The second column displays 
which sites within the unit cell acquire a sign. A ‘-1’ in the third column indicates, that the gauge transformation is ‘staggered’ for neighboring 
unit cells, i.e. it contains an additional factor (—l) k ° n . 


Character table for the point group O 


O 

E 

8 C 3 

6 C 2 ’ 

6 C 4 

3C 2 

linear functions 

quadratic functions 

cubic functions 

Ai 

T 

1 

1 

1 

1 

- 

2 1 2 1 2 

x + y + z 

- 

A 2 

1 

1 

-1 

-1 

1 

- 

- 

xyz 

E 

2 

-1 

0 

0 

2 

- 

{x 2 -y 2 ,2z 2 -x 2 -y 2 ) 

- 

Ti 

3 

0 

-1 

1 

-1 

(*£> I /5 Z )(Rx> Ry , Rz ) 

- 

(x 3 ,y 3 , z 3 )[x(z 2 + y 2 ),y(z 2 + x 2 ), z(x 2 + y 2 )\ 

t 2 

3 

0 

1 

-1 

-1 

- 

(xy,xz,yz) 

[x{z 2 - y 2 ), y(z 2 - x 2 ), z(x 2 - y 2 )\ 


TABLE III. Character table for the point group O. 



